- Physical Review Journals
All JournalsPhysics Magazine

Physical Review D

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Open Access

Resimulation-based self-supervised learning for pretraining physics foundation models

P. Harris

J. Krupa*

M. Kagan†

B. Maier‡

N. Woodward

  • *Contact author: jkrupa@mit.edu
  • †Contact author: makagan@slac.stanford.edu
  • ‡Contact author: b.maier@imperial.ac.uk
PDF

Phys. Rev. D 111, 032010 – Published 21 February, 2025

DOI: https://doi.org/10.1103/PhysRevD.111.032010

Abstract

Self-supervised learning (SSL) is at the core of training modern large machine learning models, providing a scheme for learning powerful representations that can be used in a variety of downstream tasks. However, SSL strategies must be adapted to the type of training data and downstream tasks required. We propose resimulation-based self-supervised representation learning (RS3L), a novel simulation-based SSL strategy that employs a method of resimulation to drive data augmentation for contrastive learning in the physical sciences, particularly, in fields that rely on stochastic simulators. By intervening in the middle of the simulation process and rerunning simulation components downstream of the intervention, we generate multiple realizations of an event, thus producing a set of augmentations covering all physics-driven variations available in the simulator. Using experiments from high-energy physics, we explore how this strategy may enable the development of a foundation model; we show how RS3L pretraining enables powerful performance in downstream tasks such as discrimination of a variety of objects and uncertainty mitigation. In addition to our results, we make the RS3L dataset publicly available for further studies on how to improve SSL strategies.

View figure in article

Physics Subject Headings (PhySH)

Article Text

References (59)

  1. R. Bommasani et al., On the opportunities and risks of foundation models, arXiv:2108.07258.
  2. Z. Tong, Y. Song, J. Wang, and L. Wang, Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training, arXiv:2203.12602.
  3. T. B. Brown et al., Language models are few-shot learners, arXiv:2005.14165.
  4. S. Wen Yang et al., Superb: Speech processing universal performance benchmark, arXiv:2105.01051.
  5. A. Rives et al., Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences, Proc. Natl. Acad. Sci. U.S.A. 118, e2016239118 (2021).
  6. J. Ross, B. Belgodere, and V. Chenthamarakshan, Large-scale chemical language representations capture molecular structure and properties, Nat. Mach. Intell. 4, 1256 (2022).
  7. J. Pan, Large language model for molecular chemistry, Nat. Commun. Sci. 3, 5 (2023).
  8. R. Irwin, S. Dimitriadis, J. He, and E. J. Bjerrum, Chemformer: A pre-trained transformer for computational chemistry, Mach. Learn. Sci. Technol. 3, 015022 (2022).
  9. W. Ahmad, E. Simon, S. Chithrananda, G. Grand, and B. Ramsundar, Chemberta-2: Towards chemical foundation models, arXiv:2209.01712.
  10. S. Subramanian, P. Harrington, K. Keutzer, W. Bhimji, D. Morozov, M. Mahoney, and A. Gholami, Towards foundation models for scientific machine learning: Characterizing scaling and transfer behavior, arXiv:2306.00258.
  11. M. McCabe et al., Multiple physics pretraining for physical surrogate models, arXiv:2310.02994.
  12. F. Lanusse et al., Astroclip: Cross-modal pre-training for astronomical foundation models, Mon. Not. R. Astron. Soc. 531, 4990 (2024).
  13. M. Walmsley, I. V. Slijepcevic, M. Bowles, and A. M. M. Scaife, Towards galaxy foundation models with hybrid contrastive learning, arXiv:2206.11927.
  14. J. Birk, A. Hallin, and G. Kasieczka, OmniJet-α: the first cross-task foundation model for particle physics, Mach. Learn. Sci. Technol. 5, 035031 (2024).
  15. T. Chen, S. Kornblith, M. Norouzi, and G. E. Hinton, A simple framework for contrastive learning of visual representations, arXiv:2002.05709.
  16. J. M. Campbell et al., Event generators for high-energy physics experiments, SciPost Phys. 16, 130 (2024).
  17. G. Aad et al. (ATLAS Collaboration), The ATLAS experiment at the CERN Large Hadron Collider, J. Instrum. 3, S08003 (2008).
  18. S. Chatrchyan et al. (CMS Collaboration), The CMS Experiment at the CERN LHC, J. Instrum. 3, S08004 (2008).
  19. L. Evans and P. Bryant, LHC machine, J. Instrum. 3, S08001 (2008).
  20. M. Aaboud et al. (ATLAS Collaboration), Observation of H→bb¯ decays and VH production with the ATLAS detector, Phys. Lett. B 786, 59 (2018).
  21. A. M. Sirunyan et al. (CMS Collaboration), Observation of Higgs boson decay to bottom quarks, Phys. Rev. Lett. 121, 121801 (2018).
  22. G. Aad et al. (ATLAS Collaboration), Measurement of the associated production of a Higgs boson decaying into b-quarks with a vector boson at high transverse momentum in pp collisions at s=13  TeV with the ATLAS detector, Phys. Lett. B 816, 136204 (2021).
  23. G. Aad et al. (ATLAS Collaboration), Measurements of WH and ZH production in the H→bb¯ decay channel in pp collisions at 13 TeV with the ATLAS detector, Eur. Phys. J. C 81, 178 (2021).
  24. A. Tumasyan et al. (CMS Collaboration), Measurement of simplified template cross sections of the Higgs boson produced in association with W or Z bosons in the H→bb− decay channel in proton-proton collisions at s=13  TeV, Phys. Rev. D 109, 092011 (2024).
  25. B. M. Dillon, G. Kasieczka, H. Olischlager, T. Plehn, P. Sorrenson, and L. Vogel, Symmetries, Safety, and Self-Supervision, SciPost Phys. 12, 188 (2022).
  26. E. Witkowski and D. Whiteson, Learning broken symmetries with resimulation and encouraged invariance, arXiv:2311.05952.
  27. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv:1810.04805.
  28. H. Bao, L. Dong, S. Piao, and F. Wei, Beit: Bert pre-training of image transformers, arXiv:2106.08254.
  29. L. Heinrich, T. Golling, M. Kagan, S. Klein, M. Leigh, M. Osadchy, and J. A. Raine, Masked particle modeling on sets: Towards self-supervised high energy physics foundation models, Mach. Learn. Sci. Technol. 5, 035074 (2024).
  30. T. Kishimoto, M. Morinaga, M. Saito, and J. Tanaka, Pre-training strategy using real particle collision data for event classification in collider physics, in Presented at the Machine Learning and the Physical Sciences Workshop, NeurIPS 2023 (2023), arXiv:2312.06909.
  31. M. Vigl, N. Hartman, and L. Heinrich, Finetuning foundation models for joint analysis optimization, Mach. Learn. Sci. Technol. 5, 025075 (2024).
  32. A. Akhmetzhanova, S. Mishra-Sharma, and C. Dvorkin, Data compression and inference in cosmology with self-supervised machine learning, Mon. Not. R. Astron. Soc. 527, 7459 (2024).
  33. T. Sjöstrand, S. Ask, J. R. Christiansen, R. Corke, N. Desai, P. Ilten, S. Mrenna, S. Prestel, C. O. Rasmussen, and P. Z. Skands, An introduction to PYTHIA 8.2, Comput. Phys. Commun. 191, 159 (2015).
  34. A. M. Sirunyan et al. (CMS Collaboration), Extraction and validation of a new set of CMS pythia8 tunes from underlying-event measurements, Eur. Phys. J. C 80, 4 (2020).
  35. J. Bellm et al., herwig7.1 release note, arXiv:1705.06919.
  36. J. Shlomi, P. Battaglia, and J.-R. Vlimant, Graph neural networks in particle physics, Mach. Learn. Sci. Technol. 2, 021001 (2020).
  37. J. Alwall, R. Frederix, S. Frixione, V. Hirschi, F. Maltoni, O. Mattelaer, H.-S. Shao, T. Stelzer, P. Torrielli, and M. Zaro, The automated computation of tree-level and next-to-leading order differential cross sections, and their matching to parton shower simulations, J. High Energy Phys. 07 (2014) 079.
  38. R. D. Ball, V. Bertone, S. Carrazza, C. S. Deans, L. D. Debbio, S. Forte, A. Guffanti, N. P. Hartland, J. I. Latorre, J. Rojo, and M. Ubiali, Parton distributions for the LHC run II, J. High Energy Phys. 04 (2015) 040.
  39. J. de Favereau, C. Delaere, P. Demin, A. Giammanco, V. Lemaitre, A. Mertens, and M. Selvaggi, delphes3: A modular framework for fast simulation of a generic collider experiment, J. High Energy Phys. 02 (2014) 057.
  40. M. Cacciari, G. P. Salam, and G. Soyez, The anti-kt jet clustering algorithm, J. High Energy Phys. 04 (2008) 063.
  41. Y. Wang, Y. Sun, Z. Liu, S. E. Sarma, M. M. Bronstein, and J. M. Solomon, Dynamic graph CNN for learning on point clouds, arXiv:1801.07829.
  42. J. Devlin, M. Chang, K. Lee, and K. Toutanova, BERT: Pre-training of deep bidirectional transformers for language understanding, arXiv:1810.04805.
  43. B. Maier, S. M. Narayanan, G. de Castro, M. Goncharov, C. Paus, and M. Schott, Pile-up mitigation using attention, Mach. Learn. Sci. Technol. 3, 025012 (2022).
  44. D.-A. Clevert, T. Unterthiner, and S. Hochreiter, Fast and accurate deep network learning by exponential linear units (elus), in ICLR (Poster) (2016).
  45. I. Moult, L. Necib, and J. Thaler, New angles on energy correlation functions, J. High Energy Phys. 12 (2016) 153.
  46. A. M. Sirunyan et al. (CMS Collaboration), Search for dark matter produced in association with a Higgs boson decaying to a pair of bottom quarks in proton–proton collisions at s=13  TeV, Eur. Phys. J. C 79, 280 (2019).
  47. A. M. Sirunyan et al. (CMS Collaboration), Search for dark matter particles produced in association with a Higgs boson in proton-proton collisions at s=13  TeV, J. High Energy Phys. 03 (2020) 025.
  48. A. M. Sirunyan et al. (CMS Collaboration), Inclusive search for highly boosted Higgs bosons decaying to bottom quark-antiquark pairs in proton-proton collisions at s=13  TeV, J. High Energy Phys. 12 (2020) 085.
  49. A. M. Sirunyan et al. (CMS Collaboration), Search for low mass vector resonances decaying into quark-antiquark pairs in proton-proton collisions at s=13  TeV, J. High Energy Phys. 01 (2018) 097.
  50. A. M. Sirunyan et al. (CMS Collaboration), Search for low mass vector resonances decaying into quark-antiquark pairs in proton-proton collisions at s=13  TeV, Phys. Rev. D 100, 112007 (2019).
  51. Pearson’s correlation coefficient, in Encyclopedia of Public Health, edited by W. Kirch (Springer Netherlands, Dordrecht, 2008), pp. 1090–1091.
  52. L. van der Maaten and G. Hinton, Visualizing data using t-sne, J. Mach. Learn. Res. 9, 2579 (2008).
  53. L. N. Vaseršteĭn, Markov processes over denumerable products of spaces, describing large systems of automata, Prob. Peredachi Inf. 5 (1969).
  54. L. V. Kantorovich, Mathematical methods of organizing and planning production, Manage. Sci. 6, 366 (1960).
  55. A. Ramdas, N. Garcia, and M. Cuturi, On Wasserstein two sample testing and related families of nonparametric tests, arXiv:1509.02237.
  56. A. Bardes, J. Ponce, and Y. LeCun, Vicreg: Variance-invariance-covariance regularization for self-supervised learning, arXiv:2105.04906.
  57. X. Chen and K. He, Exploring simple siamese representation learning, arXiv:2011.10566.
  58. J. Zbontar, L. Jing, I. Misra, Y. LeCun, and S. Deny, Barlow twins: Self-supervised learning via redundancy reduction, arXiv:2103.03230.
  59. https://doi.org/10.5281/zenodo.10633815

Outline

Information

Phys. Rev. D 111, 032010– Published 21 February, 2025
  • Received 29 April 2024
  • Accepted 24 January 2025
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article’s title, journal citation, and DOI. Funded by SCOAP3.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation